Skip to content

Add 16a4w_block QAT config #15878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add 16a4w_block QAT config #15878

wants to merge 1 commit into from
+201 −3

Conversation

@sxu
Copy link
Contributor

@sxu sxu commented Nov 19, 2025 •
edited
Loading

Summary: Introduce a FakeQuantizer subclass. It falls back to LPBQ observer's convert. _derived_bias_quant_spec also looks for it to correctly derive bias scale.

Open to suggestions on how to test. Naveen launched a QAT run and it seems to produce reasonable results.

Differential Revision: D87194388

@sxu sxu requested a review from cccclai as a code owner November 19, 2025 00:19
@pytorch-bot
Copy link

pytorch-bot bot commented Nov 19, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15878

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit 6dfc4ac with merge base d13c789 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 19, 2025
@meta-codesync
Copy link

meta-codesync bot commented Nov 19, 2025

@sxu has exported this pull request. If you are a Meta employee, you can view the originating Diff in D87194388.

@sxu sxu added the release notes: none Do not include this in the release notes label Nov 19, 2025
@billmguo
Copy link
Contributor

billmguo commented Nov 19, 2025

@winskuo-quic can you review and approve this diff?

@DannyYuyang-quic
Copy link
Contributor

DannyYuyang-quic commented Nov 20, 2025

I think for QAT testing we can use pseudo labels generated by the FP32 model, do a few mini training steps on the fake-quant model, and then compare outputs against the FP32 baseline (pseudo labels) within acceptable atol/rtol thresholds as usual.

@sxu sxu force-pushed the export-D87194388 branch from eb2e9f9 to 0a8fd5c Compare November 21, 2025 01:24
sxu added a commit to sxu/executorch that referenced this pull request Nov 21, 2025
@sxu @facebook-github-bot
Add 16a4w_block QAT config (pytorch#15878)
0a8fd5c 
Summary:

Introduce a FakeQuantizer subclass. It falls back to LPBQ observer's `convert`. `_derived_bias_quant_spec` also looks for it to correctly derive bias scale.

Reviewed By: viveknayakatmeta

Differential Revision: D87194388
@sxu sxu marked this pull request as draft November 21, 2025 01:36
@sxu sxu force-pushed the export-D87194388 branch from 0a8fd5c to d26dc52 Compare November 24, 2025 17:47
sxu added a commit to sxu/executorch that referenced this pull request Nov 24, 2025
@sxu @facebook-github-bot
Add 16a4w_block QAT config (pytorch#15878)
d26dc52 
Summary:

Introduce a FakeQuantizer subclass. It falls back to LPBQ observer's `convert`. `_derived_bias_quant_spec` also looks for it to correctly derive bias scale.

Reviewed By: viveknayakatmeta

Differential Revision: D87194388
@sxu
Copy link
Contributor Author

sxu commented Nov 24, 2025 •
edited
Loading

I added a test under TestQNNQuantizedOperator, however it's running to an error

File "/data/users/shenchenxu/fbsource/buck-out/v2/gen/fbcode/102760a3f8758529/executorch/backends/qualcomm/tests/fb/__test_qnn_delegate_simulator__/test_qnn_delegate_simulator#link-tree/executorch/backends/qualcomm/builders/node_visitor.py", line 472, in define_tensor
         tensor_wrapper = PyQnnWrapper.TensorWrapper(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Unable to cast Python instance of type <class 'numpy.ndarray'> to C++ type '?' (#define PYBIND11_DETAILED_ERROR_MESSAGES or compile in debug mode for details)

It seems that this is caused by the quant_configs argument passed here:

executorch/backends/qualcomm/builders/node_visitor.py

Line 475 in d13c789

quant_configs,
containing a NumPy array. However that's actually expected for block quant as the NumPy array is explicitly created here:

executorch/backends/qualcomm/builders/node_visitor.py

Line 198 in d13c789

quant_config[QCOM_BLOCK_SCALES] = torch.cat(quantized_scales).detach().numpy()
. @haowhsu-quic do you know how this is supposed to work?

Another strange thing is that this error doesn't happen when we export the real model we use (however that export script is internal and I can't share it unfortunately.

@sxu sxu marked this pull request as ready for review November 24, 2025 17:56
@sxu
Copy link
Contributor Author

sxu commented Nov 24, 2025

If I manually convert this line

executorch/backends/qualcomm/builders/node_visitor.py

Line 198 in d13c789

quant_config[QCOM_BLOCK_SCALES] = torch.cat(quantized_scales).detach().numpy()

to be a Python list it seems to work, perhaps some pybind dependencies weren't being included when running the test?

@sxu @facebook-github-bot
Add 16a4w_block QAT config (pytorch#15878)
6dfc4ac 
Summary:

Introduce a FakeQuantizer subclass. It falls back to LPBQ observer's `convert`. `_derived_bias_quant_spec` also looks for it to correctly derive bias scale.

Reviewed By: viveknayakatmeta

Differential Revision: D87194388
@sxu sxu force-pushed the export-D87194388 branch from d26dc52 to 6dfc4ac Compare November 24, 2025 18:31
@sxu
Copy link
Contributor Author

sxu commented Nov 24, 2025

Please ignore my question above about NumPy array, our internal test runner was missing a pybind dependency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cccclai cccclai Awaiting requested review from cccclai cccclai is a code owner

@shewu-quic shewu-quic Awaiting requested review from shewu-quic

@winskuo-quic winskuo-quic Awaiting requested review from winskuo-quic

1 more reviewer

@haowhsu-quic haowhsu-quic haowhsu-quic left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported meta-exported release notes: none Do not include this in the release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sxu @billmguo @DannyYuyang-quic @haowhsu-quic